Advanced Machine Learning Practical 4 : Regression ( SVR , RVR , GPR )

During this week’s practical we will focus on understanding and comparing the performance of the different regression methods seen in class, namely SVR and its variants ( -SVR, ν-SVR and its Bayesian counterpart RVR), as well as an introduction to Bayesian Linear Regression and Gaussian Process Regression (GPR). Similar to non-linear classification methods, the non-linear regression methods we will see in this tutorial (SVR/RVR/GPR) rely on a set of hyper-parameters ( : tube sensitivity, ν: bounds, σ: kernel width for RBF) which optimize the objective function (and consequently the parameters of the regressive function). Choosing the best hyperparameters for your dataset is not a trivial task, we will analyze their effect on the regression accuracy and how to choose an admissible range of parameters. A standard way of finding these optimal hyper-parameters is by doing a grid search, i.e. systematically evaluating each possible combination of parameters within a given range. We will do this for different datasets and discuss the difference in performance, model complexity and sensitivity to hyper-parameters.